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Abstract: This paper presents a framework for evaluating efforts by organizations to share research 
as exemplified on their websites, then reports the result of an evaluation of these efforts in 100 
organizations. The result shows that the overall research sharing efforts of these organizations are 
modest and not well aligned to evidence on effective practice in this area. Organizations tend to 
devote more efforts to making products available while interpersonal strategies, though more 
effective, are less used. Most efforts involve one way communication to potential users. The paper 
concludes with suggestions for practice and for further research. 
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Evaluacion de los esfuerzos de organization para difundir resultados de las Investigaciones 
en Education 

Resumen: En este trabajo se presenta un marco para evaluar los esfuerzos de organizaciones para 
difundir la investigation, como se ejemplifica en sus sitios web, y luego presenta los resultados de 
una evaluacion de estos esfuerzos en 100 organizaciones. Los resultados muestran que en general los 
esfuerzos de estas organizaciones para compartir la investigation son modestos y no muy bien 
alineados con las evidencias acerca de las practicas efectivas en este ambito. Las organizaciones 
tienden a dedicar mas esfuerzos para hacer productos disponibles mientras que las estrategias 
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interpersonales son mas eficaces, aunque menos utilizadas. La mayoria de los esfuerzos implican una 
forma de comunicacion unidirectional a los usuarios potenciales. El articulo concluye con 
sugerencias para mejorar esas practicas y futuras investigaciones. 

Palabras clave: movilizacion de conocimientos; difusion de resultados de investigation. 

Avalia§ao dos Esforgos Organizational para a difundir resultados de Pesquisas em 
Educa§ao 

Resumo: Este artigo apresenta uma estrutura para avaliar os esforyos de organizagoes para partilhar 
pesquisa, como exemplificado nos seus websites, de seguida, relata os resultados de uma avaliagao 
desses esforgos em 100 organizagSes. Os resultados mostram que globalmente os esforyos dessas 
organizagoes para a partilha de pesquisa sao modestos e nao muito bem alinhados com evidencias 
sobre praticas efetivas nesta area. As organizayoes tendem a dedicar mais esforyos para tornarem os 
produtos disponiveis enquanto as estrategias interpessoais embora mais efetivas sao menos 
utilizadas. A maioria dos esforgos envolve uma forma de comunicagao unidirecional para potenciais 
usuarios. O artigo conclui com sugestoes para a pratica e para futuras pesquisas. 

Palavras-chave: mobilizacao de conhecimento; divulgacao de pesquisa. 

Introduction 


Improved communication 1 of research findings beyond the scholarly community is an 
important and growing focus of concern in all disciplines. The growing interest in this work around 
the world is motivated in part by the call for greater accountability in research investments (e.g. 
Shavelson & Towne, 2002; British Academy, 2008), but even more by growing appreciation of the 
importance of research for enhancing the quality of public services such as health, justice, and 
education (Levin, 2004, 2010; Nutley, Walter & Davies, 2007). 

Although many different terms 2 have been used to refer to the various processes that 
connect research to policy and practice, we refer to this enterprise as ‘knowledge mobilization’ [KM] 
because that phrase well reflects the intentional and active elements required to make knowledge 
relevant to policy and practice (Cooper, Levin & Campbell, 2009). We also recognize, as described 
well by Nutley et al. (2007), that there are different meanings ascribed to all the key terms involved. 
For example, people differ on what counts as ‘research’, while most analysts also recognize that the 
‘use’ of research also has multiple dimensions, from changing what people know to affecting how 
people think to influencing how they work (Weiss, 1979, Knott & Wildavsky, 1980, Nutley, Walter 
& Davies, 2007). Ideas of mobilizing or using research knowledge are also contested politically, with 
some worrying that this movement is really intended to narrow and control professional work (e.g. 
Ball, 2001). Yet despite these debates, virtually everyone would be in favor of stronger connections 
between research and practice in some ways, even if there is disagreement about how those 
connections should work. 

Our conceptual model for research knowledge mobilization is drawn from Levin (2011) and 
is shown in Figure 1. It draws attention to 3 connected ‘contexts’ - the production of research, the 
organizations or settings where it may be used, and the various bodies or processes that mediate 
between research production and use. These ‘contexts’ are not necessarily synonymous with 


1 In this paper we use interchangeably terms such as ‘dissemination’, ‘sharing’ and ‘communicating’ research. 
We discuss these conceptual issues later in the paper. 

2 For example knowledge transfer, knowledge translation, research dissemination, or research utilization. 
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particular organizations, since many organizations are involved in two or even all three of these 
functions. 
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The triangles represent functions, not necessarily structures. Some people or groups operate in more than one context. Arrows 
represent strength of relationships. ICM occurs where two or more of these contexts or functions interact 


Figure 1. Research Knowledge Mobilisation 


Many different kinds of organizations are involved in attempting to mobilize research 
knowledge. These include organizations with primary interests in each of the three areas, or in all 
three - from universities to school systems to policy bodies to a whole range of intermediary 
organizations such as think tanks, unions, and lobby groups (Cooper, 2012). Active involvement in 
doing research is not a requirement for active engagement in mobilizing research knowledge. 

In practice, a wide range of knowledge mobilization efforts can be found in these various 
organizations. Nutley, Walter and Davis (2009) define five kinds of research communication 
activities: tailored dissemination, interaction, social influence, facilitation, and reminders and 
incentives. Each of these categories can include a range of specific actions, any or all of which can 
be employed in different ways by various organizations. There are many case studies of these efforts 
and occasional evaluations of them (the journal Evidence & Policy is one source of many examples). 
Yet at an aggregate level little is known about the ways that different kinds of organizations attempt 
to mobilize research knowledge. 

Decades ago, Klein & Gwaltney (1991) had already described three types of research 
dissemination. The first type they call spread, meaning one-way information distribution. Most 
research publications and other audio and visual materials belong to this type. Within the category, 
syntheses and interpretations of research knowledge can be viewed as preferable to the reporting of 
individual studies. The second category they call choice, which focuses on helping knowledge users 
acquire different sources of information by providing more options. If the first type is passive in 
nature, the second type is more reactive and responsive. The third type of dissemination is exchange, 
which centers on interactions between and among people and organizations and emphasizes the 
“multidirectional flow of information” (Klein & Gwaltney, 1991, p.246) through networks, feedback 
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system, and so on. Many professional and governmental organizations in education have used this 
approach to build and maintain networks to better support use of research: “It is viewed as 
interactive dissemination and is congment with recent education trends such as restructuring schools 
to empower teachers and students and accountability goals to identify and share what works” (Klein 
& Gwaltney, 1991, p.246). 

As an illustration of how ideas recycle across time, the Klein & Gwaltney typology is very 
similar to a more recent typology of producer push, user pull and exchange (Lavis et al., 2003) or a 
very similar proposed by Amara et al. (2004). Moreover, all these conceptualizations fit well with the 
basic model as shown in Figure 1. 

The World Wide Web and Research Communication 

It is no exaggeration to say that approaches to research communication have been 
transformed by the development of the World Wide Web. The web makes vast amounts of 
information available to every user and has become the primary vehicle for information sharing in 
almost every field of activity (Chavkin & Chavkin, 2008; Lederbogen & Trebbe, 2003). Whereas 
twenty or thirty years ago the challenge was to find research material, much of which was not readily 
accessible, the situation is now reversed, and the main challenge is to sort out what information is 
valid and reliable from the masses available. For example, a study by our team (suppressed for blind 
review) has found that in almost every category of education there are literally thousands of 
resources readily available through the internet claiming to be based on or informed by research. 

Internet technologies “have opened the door to an additional and much broader range of 
dissemination possibilities and have generated entirely new forms of content that must be shared” 
(Association of American Universities et al., 2009, p. 2). These include not only new forms of 
product dissemination, but also much more interactive strategies (Lee & Garvin, 2003; Nutley, et al., 
2007). In recent years social media are further changing research communication in important ways, 
though that development is not a focus of this paper. 

For most people, the web is the now the first and most important way to look for research 
knowledge. Moreover, some web embedded features such as electronic storage and search engines 
create opportunities for organizations to document, arrange and present research knowledge with 
“extraordinary flexibility, depth, and power” (Association of American Universities, et al., 2009, p.3) 
and potentially reach more audiences at relatively low cost. 

For purpose of research dissemination, websites are essential components of the 
information architecture in organizations (Baker, 2005). Institutions of all kinds are putting 
considerable effort into creating, maintaining, and promoting websites. In the developed world 
virtually every organization involved in research communication now has its own website, often 
quite extensive, and many organizations in middle and lower income countries also have a web 
presence. In the case of universities, there may be many sites as sub-units such as faculties and 
centers, and even individual faculty members maintain their own sites. 

Websites offer many features for organizations to share their knowledge such as inventories 
of research that can be structured in multiple ways. All kinds of materials including print, video, 
audio or presentation materials can be organized and posted on the web. Researchers may offer 
tutorials about their own work. Institutions sponsor web-casts of academic conferences, archives of 
seminars, lectures and so on. Moreover, search engines are powerful tools for users to locate 
information. A website is also a platform both to support and supplement various other 
dissemination practices (Supyuenyong, et al., 2009). Specifically, the web allows many new 
communication vehicles such as bulletin boards, audio and video conferencing, listservs, wikis, social 
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media, feeds and others which collectively are permanently changing the nature of information 
dissemination. 

However, so far, little empirical work has been done concerning research communication on 
institutional websites. This project is intended to address this gap in the literature by focusing on the 
research sharing practices of organizations based on the activities shown on their websites. The 
work explores the way websites are and could be used for the purpose of research communication, 
as well as finding patterns of research dissemination strategies across institutions. Because there was 
no instrument available to analyze these practices, we developed a system of indicators through 
which institutional websites can be understood, measured, compared and improved. 

Of course, websites do not necessarily contain or reflect all of an organization’s 
dissemination practices; other research dissemination activities could be going on that are not 
reported on the web. Yet given their centrality to this work, websites do provide much relevant 
information (Selvanathan, 2007). It therefore seems reasonable to analyze organizational websites as 
a way of understanding their strategies for research dissemination. It is also important to note that 
this project was not aimed at assessing the quality of the websites themselves, such as their aesthetics 
or functionality; its focus is solely on what they tell us about overall knowledge mobilization and 
research dissemination activities of institutions. 

Method 

We began by looking at research dissemination activities on the websites of a large number 
of organizations. Our search was deliberately broad, starting with educational institutions in the 
various parts of our conceptual framework such as universities, education ministries, school districts, 
and other education stakeholders. For comparative purposes we also looked at organizations 
beyond the education sector. Because the health sector has done the most work on mobilizing 
research knowledge, we looked at a number of health organizations, and because our model draws 
attention to the importance of intermediary organizations in the dissemination of research, we also 
looked at several organizations of that type. As a Canadian research team we focused on Canadian 
organizations but also looked at organizations in other countries for comparative purposes. Our 
choice of organizations included systematic elements, such as looking at most or all education 
ministries in Canada, or a selection of the largest school districts, but we also considered 
organizations that we had heard about in various ways, or that were linked to some of the sites we 
initially reviewed. 

The analyses were done by a team of graduate students and faculty. We began with an open 
and subjective team discussion of what we regarded as good dissemination activity on various sites. 
Several rounds of such discussion, coupled with the reading we were doing in the literature on 
knowledge mobilization and research dissemination, led us to develop a common set of criteria for 
evaluating the sites, through which each site was awarded points for various features. 

Evaluation Framework 

Over time we added elements to our evaluative frame. As each version of the scoring system 
was developed, members of the team re-evaluated sites using the new criteria. In each round, some 
sites were evaluated by at least two team members or at team meetings so we could judge the 
reliability of the ratings. At team meetings we would discuss whether the new ratings captured all the 
important features, or whether the criteria for ratings were sufficiently clear. We also continued 
reading the literature on research mobilization so that our analysis could be informed by current 
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evidence. Over several months of such evaluation, the system was steadily refined. Some categories 
were clarified, other subdivided, and in other cases the criteria within the categories were made more 
specific. Our current formulation is in Figure 2 below. 



Figure 2. Conceptual Framework for Evaluating Knowledge Mobilisation on Organisation Websites 


The three boxes on the left represent the three major approaches to research mobilization, 
inductively derived (but also, though at the time unknown to us, very similar to Klein & Gwaltney, 
1991), while the categories inside the right hand box are quality indicators for each approach. We 
explain each of these categories in more detail. 

Products are attempts to communicate knowledge in written or audiovisual forms, such as 
journal articles, reports, books, syntheses, bulletins, research newsletters, or video or audio clips. 
Events are activities such as lectures, conferences, seminars, workshops, symposia, or exhibitions 
where the aim is to share research with practitioners and users. As such, this category excludes 
events that are primarily focused on the exchange of research findings within the research 
community, such as academic conferences. Events in this category can be real or virtual. 

Networks refer to efforts to build lasting relationships among and between knowledge producers 
and potential users. The primary interest is in connections that actively share knowledge between 
researchers and target audiences in a way that is sustained over time and highly interactive (reference 
removed for blind review). Again, the focus is on interaction that extends outside the research 
community itself. 

The first category is relatively self-explanatory. Often, research communication is equated 
with products of different kinds such as articles or reports. The range of such products can be 
enormous, from larger reports to executive summaries, to syntheses, research briefs, and a variety of 
products written for particular audiences. More recently, in conjunction with the capacities of the 
web, research products have included audio and video materials, blogs, podcasts and even more 
recently the use of social media. 
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The use of events to communicate research findings is also longstanding. Many professional 
development activities, in many disciplines, could be considered as research communication through 
events. 

While both these categories of activity have merit, empirical work has made it increasingly 
clear that the active take-up of research findings requires more than either products or events. From 
a variety of perspectives, ongoing interaction has been shown to be the most powerful factor in 
shaping people’s acceptance and use of research evidence (Nutley et al., 2007; Hemsley-Brown & 
Sharp, 2003; Wenger, 1998; Masuda et al., 2009). For example, Lavis et al. (2005) highlight the 
interaction between researchers and research users as follows: 

Research on the transfer of research knowledge to and its uptake by managerial and policy 
audiences has demonstrated that interaction between researchers and these audiences (or 
representative members of these audiences) appears to be important to explaining why some 
types of research knowledge are used and not others (p.226). 

Flowever interaction need not just be between researchers and others. Since the strongest 
influence on professional practice is the views of peers (Mitton et al., 2007), interaction among 
professionals is also a powerful vehicle for mobilizing research knowledge, [reference suppressed] 
emphasize that “collaborative projects and networking are vital to KM [knowledge mobilization] as 
system improvement depends on the multiple contributions among different stakeholder groups” 
(p.355). 

Mathiassen and Vogelsang (2005) distinguish between networks and networking. The former 
stresses the structure while the latter term emphasizes relationship building in terms of trust and 
collaboration. To assess relationships, we could consider their content, form and intensity. Content 
here includes services, information, or even emotions that are generated through interpersonal 
contact. Form refers to how long these relationships last and how close these relationships are. 
Intensity considers how frequently the information exchange happens (Chen & Mohamed, 2007). 

Within these three categories of KM work, as our analysis proceeded, we began to identify 
particular elements that might be predictive of greater effectiveness. Those eventually became the 
four categories in the right hand box in Figure 2. 

Different Types of Products, Events and Networks 

Empirical research on knowledge mobilization (Cordingley, 2008; Landry et al., 2001) 
suggests that the use of multiple types of products, events and networks can effectively promote 
research utilization. Thus, the use of multiple types of KM strategies by organizations is one 
indicator for assessing the overall KM performance. 

Although networks are important, they are not automatically valuable. Williams & Bailey 
(2002) stress that it is essential to differentiate different types of networks to study their dynamics 
and provide appropriate support, as the concept itself is used indiscriminately in the current 
literature. Different types of networks are emerging such as knowledge networks (Clark, 1998), 
innovation networks (Pittaway et al, 2004), research networks (Gunn, 2002, Williams & Bailey, 

2002), and so on. Some networks are intended to disseminate knowledge to the wider public, while 
some networks are limited to the communication among their members. 

Assessing the value of networks requires knowledge about their characteristics, functions 
and purposes as well as the degree of interactivity and frequency of communication. In this project, 
we considered three types: internal networks, external vertical networks, and external horizontal 
networks. Internal networks (Creech, 2005) refer to informal or structured network of people in 
organizations with the purpose of sharing information to achieve KM objectives. External vertical 
networks (Fliaster, 2008) indicate that the organization has a clear KM-focused mandate in working 
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with other institutions. External horizontal networks (Fliaster, 2008) describes networks made up of 
people within and across fields. 

Focus of Audience 

Research utilization literature (e.g. Nutley et al., 2007; Mitton et al., 2007) tends to divide 
potential users into three categories: policy makers, or those who must make decisions about 
resource allocations, program support, or new legislation and regulations; citizens, or those who are 
consumers of the services or who may otherwise be affected by government policies; and service 
providers, or those who are involved in the operation of actual services-e.g., schools, police services, 
health facilities, and social service programs (Yin, et al., 1981, p.557). 

In any strategy it is important to think about approaches tailored to different audiences, and 
about giving easy access to the main findings or conclusions (Ruzek & Rosen, 2009). Each of these 
groups may need different kinds of products, events and networks. Simply listing all research 
reports, or giving a schedule of activities will not suffice because users usually do not have the ability 
to evaluate the applicability of the information (Davenport & Volpel, 2001). A clear indication of 
the intended audience is an important measurement to predict the uptake of research. 

Accessibility 

Accessibility has long been recognized as a major impediment to effective research 
knowledge mobilization (Hemsley-Brown & Sharp, 2004; Nutley, Walter & Davies, 2007). As 
noted, the internet provides enormous new possibilities for making research more accessible to 
users. Features such as links, search engines, and menus bars embedded in websites enable web 
users to be more active in obtaining information. On-line interaction makes possible much more 
synergy between information senders and receivers (Tremayne & Dunwoody, 2001). Knowledge 
mobilization requires that resources be available to users, hence organizations which freely provide 
products, events and network score points across this indicator, while those requiring pay or 
registration get fewer points. We also evaluate the readability (language level) of the relevant 
information on the website, using the Flesch-Kincaid reading level assessment of the mission 
statement of each organization, on the grounds that esoteric language can also be a barrier to 
effective communication. The readability test produces a score based on the number of words per 
sentence and the number of syllables per word. 

Ease of Use 

Ease of use refers to how much effort is required to find and use research information. 
Assessing this criterion varies based on the nature of products, events and networks. Feedback 
mechanisms to allow input from potential users, especially if interactive, are one essential element to 
improve KM activities. Such mechanisms include the opportunity to respond to materials, to post 
queries or comments, or to participate in an going discussion forum or listserv. However simply 
having a “contact us” feature, such seen on almost every institutional website, is not included here 
because this does not provide real interaction. 

Frequency of interaction assesses how frequently those involved in a network interact with 
each other, which is one indicator of the efforts by an organization to reach out to its audiences 
rather than passively waiting for them to seek out what they need. Features of this criterion include 
follow ups after certain KM events, posting a summary or a visual record of the events, setting up 
various forms of archives for products and information from previous events, searchable directories 
for networks, events calendars, and so on. All these features help users get the information they 
need in a relatively short time. 
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Additional Elements 

It is also worth noting that the three strategies of products, events and networks require a 
balance. Multiple and varied strategies seem to be essential toed reach different audiences and 
increase the impact of research. For example, organizations can use various types of products 
(various forms of print, audio, video), or of events (seminars, workshops, longer and shorter) and 
networks (short-term or task focused vs long term or more general). We therefore award additional 
points for balance within and across strategies. Finally, we identified another element that we 
considered important but was not part of the three main strategies; this was a clear and prominent 
statement on the site of the organization’s intentions and focus around research mobilization as a 
clear statement to visitors of the value of this work. 

The system that we developed on this basis and used to rate sites has a total potential score 
of 72 points (Table 1). In keeping with the research on active vs passive dissemination measures, 
more points (20 points each) are allocated for events and network, with fewer points (12 points) for 
products. 

Results 

Over 100 institutional websites have been scored using this system. This sample covers 
different types of organizations (intermediaries, governments, universities, faculties of education, 
etc.) in different sectors (education, health, etc.). The single largest category is faculties of education 
in universities, but there are also significant numbers of sites of school districts, ministries of 
education, universities as a whole, and various other organizations involved in the sharing of 
research. While most are in Canada, the full set of organizations is international, though almost 
entirely from English-speaking and richer countries. The full list of the organizations scored is in 
Appendix A. As noted earlier, the set of organizations was determined by starting with Canadian 
education organizations involved in the 3 contexts of our conceptual framework and then adding 
more organizations of similar types from other sectors or countries. However the sample for this 
study has not been drawn in a way that can be claimed to be representative of a broader universe of 
institutional websites, so the findings should be regarded as preliminary and indicative. 

Scoring consisted of reviewing multiple pages on each website to determine which of these 
features was present and to what degree, a procedure that typically took 15-20 minutes per site. Each 
rater had to look for each feature on each site. For example, rating ‘different types’ of KM vehicles 
often required checking in several parts of a website since products, events and networks could well 
be located in different places. 

Several procedures were employed to maximize the reliability of the scoring process and 
results. All sites were scored by project team members who participated in developing the 
instmment and are familiar with its aims and elements. Team members, who were all doctoral 
students at the Ontario Institute for Studies in Education, also participated in several sessions to 
assess the reliability of the instrument and make improvements through comparing scores for the 
same sites and resolving disagreements. Although eight or so students (and the faculty supervisor) 
were involved in rating at least some sites, most of the rating was done by 3 students so that they 
could acquire significant competence. A random selection of sites from the sample were rated by at 
least two team members and the average scores of these sites are used for the final analysis. A formal 
interrater reliability analysis showed an IRR of .799. 
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Table 1. 

Analysis of Knowledge Mobilisation on Websites — Scoring Rubric 


Strategies 

Products 

Events 

Networks 

Overall Features 

Total 

Indicator 





Point 

Different types 

1 (1-2 types) 

2 (1-2 types) 

2 (1 type) 

1 (1 strategy) 

20 


2 (3-4 types) 

4 (3-4 types) 

4 (2 types) 

2 (2 strategies) 



3 (5-6 types) 

6 (5-6 types) 

6 (3 types) 

3 (2 strategies with 
a good balance) 

4 (three strategies) 

5 (three strategies 
with a good 
balance) 


Ease of use 

1 (means to 

2 (some 2 (events 

2 (archived 

1 (1 searching tool) 

14 


comment on 

follow-ups) archived) 

network 

2 (2 searching 



the main page) 


communication 

tools) 



2 (means to 

4 (regular 

) 

3 (3 searching 



comment on 

follow-ups) 


tools) 



specific 



4 (4 searching 



products) 



tools) 


Accessibility 

1 (small 

2 (conditions attached) 

2 (once every 

1 (low readability) 

16 


portion) 

4 (part of events with 

three month or 

2 (average 



2 (large portion) 

conditions attached) 

less) 

readability) 



3 (all) 

6 (no conditions attached) 

4 (more 
frequently) 

3 (high readability) 


Focus of 

1 (part of 

0 (no introduction) 

2 (general 


10 

Audience 

products) 

2 (brief introduction) 

introduction 




2 (part of 


about who is 




products with 


involved) 




clear application 


4 (clear about 




information) 


who is 




3 (all products) 


involved, 




4s (all products 


responsibilities 




with clear 


and purpose of 




application 

information) 


the network) 



Extra 



Collaborative 

Explicit KM 

12 

Indicators 



nature of the 

statement on the 





network 

site 





2 (some 

2 general 





indication) 

statement 





4 (strong 

4 clear statement 





indication) 

without overall 






plan 

6- strong 
statement loosely 
connected to 






overall plan 

8 strong statement 
directly connected 
to overall plan 


Totals 

12 

20 

20 

20 

72 
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The histogram in Figure 4 shows the distribution of the total scores of all the websites in our 
sample. The highest possible score is 72. A more detailed understanding comes from looking at the 
scores on some of the sub scales. Table 2 shows the average score and % of possible points in each 
category. 


Distribution of Scores 


40 

35 



0 to 10 11 to 20 21 to 30 31 to 40 41 to 50 51 to 60 


Figure 4. Distribution of Total Scores 
Table 2. 

Scores of the Sample Organisations 



Products 

(12) 

Mean Score/ 

Overall 

Percentage 

Events (20) 

Mean Score/ 

Overall Percentage 

Networks 

(20) 

Mean 

Score/ 

Overall 

Percentage 

Overall (20) 
Mean 

Score/ 

Overall 

Percentage 

Totals for 

Indicator 

(72) 

Mean 

Score/ 

Overall 

Percentage 

Different 

Types 

1.74/58% 

2.25/38% 

2.29/38% 

2.88/58% 

9.12/46% 

Ease of Use 

0.52/26% 

0.69/17% 0.90/45% 

0.19/10% 

2.32/58% 

4.58/33% 

Accessibility 

2.03/68% 

2.28/38% 

0.37/9% 

1.66/55% 

6.31/39% 

Focus of 
Audience 

1.50/38% 

1.24/62% 

1.42/35% 


4.10/41% 

Extra 

Indicators 

X 

X 

0.37/9% 

3.64/46% 

4.00/33% 

Totals for 
Each 

Strategy 

5.78/48% 

7.30/37% 

4.55/23% 

10.48/52% 

28.11/39% 
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Several conclusions arise from the analysis. First, the overall research sharing efforts of these 
organizations based on this assessment are fairly low, although virtually all organizations are 
embodying some KM features. The highest scores awarded are from 51 to 60, or from 70-85%; 
these scores were mostly for websites of faculties of education or universities. The average score is 
28, or well under 50%. This reflects our observational understanding from looking at all these sites 
that on the whole KM practices, at least as revealed on websites, are not very strong in most 
organizations. This finding is also consistent with a review of KM practices of 18 major research 
universities based on interviews with university leaders (Sa et al., 2011). Of course we did see some 
outstanding practices here and there but rarely consistently either in a particular category or for a 
particular organization. 

To be sure, the institutions in our sample vary significantly in nature and purpose. One 
would not expect to see the same KM activities in a research producing organization such as a 
university as one would in a user organization such as a school district or in an intermediary body 
such as a think tank. However each of these organizations could have a knowledge mobilization 
strategy and even with their different missions these strategies could usefully involve the range of 
criteria in our evaluative system. The products or events or networks may well differ given different 
purposes, but any kind of educational organization with a genuine desire to do a better job of 
sharing and using research evidence should use some combination of products, events and 
networks. 

Instead, we find that based on our data 54% of the organizations have no space for 
feedbacks or comments on the site while only 6% have space for comments related to specific 
research products; only 38% of sites offer all their research products or services on line free of 
charge; 29% have no research-related events and of those that do fewer than half show any follow¬ 
up for their events; about 65% of sites have some form of network related to research but of most 
do not indicate who is in the network, nor do they provide ready online access to the network’s past 
activities; about % of the sites have a clear statement of intent or purpose around their research 
mobilization work. 

Out of the three major research sharing strategies (products, events and networks), 
organizations tend to devote more efforts to products, although even here, the scores are not 
particularly high on average. Few organizations seem to move beyond disseminating research reports 
or journal articles to, say, tailoring clear messages for potential knowledge users. Even though many 
institutions offer a wide variety of research products on their websites with easy access, they quite 
often distribute them in a simple table format or in a long list, without distinguishing clearly among 
different content for the sake of users. On most sites it is hard to know what products are available, 
or what they contain, without investigating them one at a time. Sites with large numbers of products 
often do not do much to help visitors sort them out by focus or approach. 

Events and networks strategies are less used overall and when used are weaker in terms of 
ease of use, accessibility and multiple types. Events are used more than networks, and average scores 
for events are higher than for networks, reflecting, perhaps, the greater complexity of organizing 
networks. Even where these network strategies have clear instructions about purpose and content, 
they usually have conditions attached for public participation. Follow-ups such as posting material 
after advertised events are rare. 

Overall, the strategies evidenced on the websites tend to be quite passive in terms of 
research sharing, relying on visitors coming to find material rather than institutions actively working 
to mobilize knowledge. In making this claim, it should be noted that there may be further 
substantive events and networks that are not shown on websites. However from the data in this 
study, institutions generally do not display clear strategies for engaging their target audience. Where 
websites do contain vehicles such as listservs these are often not geared towards discussion of 
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research. For networks that are research related, there is often little or no introduction as to what the 
network is, how it operates or who should participate. There are few opportunities offered on sites 
for users to provide feedback of any kind, and even fewer for feedback on specific research 
products, events or networks. This reduces the likelihood of visitors engaging actively with the 
resources on the site. 

In summary, our analysis, consistent with other work cited earlier, suggests that most 
organizations are not very deliberate in their knowledge mobilization strategies. They have activities 
but not an overall strategy. As in many areas of organizational life, people tend to do what is easy, 
consistent with current practice, or being done by others rather than thinking through how goals can 
be achieved most effectively. Our future analysis will include disaggregating scores according to 
sector or geographically to see how KM strategies and indicators vary according to context. We also 
plan to score more organizations and to investigate those organizations with the highest scores to 
understand their practices more fully. 

Discussion and Further Research 

The importance of institutional websites for knowledge mobilization is increasingly 
recognized in the research literature. Websites play a vital role in well designed KM strategies (Levin, 
2011). We conclude this paper with suggestions for further research and for policy. 

From a research perspective, our work points to the need to develop new tools and 
measures related to KM in order to get better data on effects. Research sharing has many properties 
which make it difficult to study (Levin, 2011), one of which is the extensive use of self-report 
measures. The measurement tool we have developed is certainly not perfect, but it offers a starting 
point for others to examine KM work in a more objective way, and the process we used to build it 
also provides an example of how other kinds of analytic tools can be created to assess the KM work 
of various organizations. 

This research is clearly only a beginning. Different audiences will use research information 
in varying ways. Our system includes the focus of audience as an important indicator but does not 
address the varying interests and approaches of different audiences. Much detailed descriptive work 
is needed to understand how different kinds of visitors - for example teachers as compared with 
parents - use the resources provided by institutional websites and their preferences in terms of 
research format, type of language, search patterns, and so on, as well as how these different 
strategies impact learning and ultimate knowledge use. Our team is conducting other research in 
these areas and on the use of web-based research materials using web analytics and longitudinal 
survey data to assess the actual use of these materials (see www.oise.utoronto.ca/rspe for details). 

The measurement system and data presented in this paper contribute to thinking about how 
KM strategies can be more effective and how organizations can build knowledge sharing 
infrastructure that is more responsive to knowledge users. From a policy perspective, this work 
shows the importance of greater attention to research communication. If research is to have its full 
effect in helping improve education, more active measures will have to be taken to link research 
knowledge with the mechanisms that lead to changes in practice. While that discussion requires a 
paper in its own right (see Levin, 2010, 2011), the data here show clearly that current efforts are not 
optimal. 

Further investigation is needed as to whether these various strategies increase not only access 
to research but also its use in various policy and practice settings. This is itself a challenging task 
given the complexities around the meaning of ‘research use’ (Nutley et al., 2007). That being said, 
we already know enough to make some recommendations as to how organizations could change 
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their KM work to promote knowledge use. Our major recommendations in this regard would 
include organizations should be more strategic about KM work. The different components of KM 
should be integrated so that they reflect an overall strategy rather than a series of discrete actions, 
and so that the various elements are more consistent with the growing knowledge base on what is 
effective. Interpersonal and organizational links are more important than the passive provision of 
research reports or other products, since take-up is more strongly affected by social factors. This 
means more attention to building connections rather than just supplying information. Websites need 
to be linked to more active or ‘push’ strategies that make use of existing organizational mechanisms 
and processes. Organizations should track the impact of their efforts through such vehicles as web 
analytics and surveys of visitors. 

Fortunately, there is growing interest internationally in empirical research into research 
utilization, suggesting that our understanding of useful practices around knowledge mobilization is 
likely to grow, leading to more benefit from research in education and other fields. 
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Appendix A 


The full list of the sample organisations 


Organization 


URL 


Harvard Graduate School of Education 

Cambridge Faculty of Education 

Canadian Council on Learning 

Cochrane Collaboration 

University of London- institute of education 

University of Toronto -OISE 

U.S. Department of Education 

Canadian health service research foundation 

Economic and Social Research Council 

Simon Fraser - Faculty of Education 

NJ Dept, of Education 

Wallace Foundation 

the future of children 

Agency for health care research and quality 
Society for the Advancement of Excellence in 
Education 

Warwick Faculty of Education 
University of Regina - Faculty of Education 
Knowledge Loom 

University of Aarhus: Faculty of education 
Nottingham school of education 
Natural Sciences and Engineering Research 
Council of Canada (NSERC) 

Teacher’s College, Columbia University 
York University - Faculty of Education 
The Association of Education Researchers of 
Ontario 

Campbell Collaboration 
Canadian centre for knowledge mobilization 
National Centre for the dissemination of 
disability research 

Hamilton Wentworth District School Board 
Evidence for Policy and Practice Info. Centre 
Queen’s University - Faculty of Education 
University of Manitoba — Faculty of Education 
Winnipeg Regional Health Authority 
The Australian Association for Research in 
Education 

The federal ministry of education & research, 
Germany 

Durham School of Education 

Ministry of Health Services — British Columbia 

UBC- Faculty of Education 


http://www.gse.harvard.edu/ 

http://www.educ.cam.ac.uk/ 

http://www.ccl-cca.ca/CCL/Home?Language=EN 

http://www.cochrane.org/ 

http://www.ioe.ac.uk/newsEvents.html 

http://www.oise.utoronto.ca/ oise/ 

http://www.ed.gov/ 

http://www.chsrf.ca/ 

http://www.esrc.ac.uk/ESRCInfoCentre/index.aspx 

http://www.educ.sfu.ca/ 

http://www.state.nj.us/ education/ 

http:/ /www.wallacefoundation.org/Pages/ default.aspx 

http:/ / futureofchildren.org/ 

http://www.ahrq.gov/ 

http://www.saee.ca/ 

http://www2.warwick.ac.uk/fac / soc/wie/ 

http://education.uregina.ca/ 
http://knowledgeloom.org/index.isp 

http://www.dpu.dk/site.aspx?p=6515 

http://www.nottingham.ac.uk/Education/index.aspx 

http://www.nserc-crsng.gc.ca/ 

http://www. tc.columbia.edu/ 

http:/ / edu.yorku.ca/index.html 

http://www.aero-ontario.org/ioomla/ 

http: / / www.campbellcollaboration.org/ 

http://www.cckm.ca/ 

http://www.fastuk.org/atcommunity/orgview.php?id=1700 

http://www.hwdsb.on.ca/ 

http://eppi.ioe.ac.uk/ cms/ 
http:/ / educ.queensu.ca/index.html 
http://www.umanitoba.ca/faculties/education/ 

http://www.wrha.mb.ca/ 

http://www.aare.edu.au/live/index.php? 

http://www.bmbf.de/en/index.php 

http://www.dur.ac.uk/education/ 

http://www.gov.bc.ca/health/ 

http:/ / educ.ubc.ca/ 
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Canadian Institute for health informatics 
Bath Faculty of Education 
Manchester school of education 
Promoting Relationships and Eliminating 
Violence 

Oxford Faculty of Education 
World Bank 
Research in practice 
ERIC Institute 

Nanyong Technological university-national 
institute of education 
Social Sciences and Humanities Research 
Council of Canada 

Australia Council for Education Research 

C.D. Howe Institute 

Society for Quality Education 

Capital Health -Edmonton 

Princeton University-Education Research 

Section 

Health Canada 

Bristol graduate school of education 
Calgary Board of Education 
Wisconsin Centre for Education Research 
Edmonton Public Schools 
Canadian Education Association 
Mathematica Policy Research 
Canadian Society for the Study of Education 
Fraser Institute 

Provincial health authority of British Columbia 

Toronto District School Board 

Rainbow District School Board 

Ministry of Health — Ontario 

Manitoba Education Research Network 

(MERN) 

Winnipeg School Division 

Thames Valley District School Board 

Saskatoon Public Schools 

Regina Public Schools 

National Union of Teachers (UK) 

Alberta Health and Wellness 

Peel Region district School Board 

Vancouver School Board 

York Region School Board 

South East Regional Health Authority - New 

Bmnswick 

What works clearinghouse 

Prince Edward Island - Education Ministry 

Department of Health - Nova Scotia 


http: / /enwildpedia.org/wiki/Canadian Institute 

http://www.bath.ac.uk/education/ 
http: / / www.education.manchester.ac.uk/ 

http:/ /prevnet.ca/ 

http: / / www.education.ox.ac.uk/home/ 

http://www worldbank.org/ 

http://www.rip.org.uk/ 

http://www.eric.ed.gov/ 

http://leo.nie.edu.sg/nieweb/programmes/load.do?id=Foundation 

http://www.sshrc.ca/site/home-accueil-eng.aspx 

http://www.acer.edu.au/l/?/news 

http://www.cdhowe.org/ 

http://www.societyforqualityeducation.org/ 

http://www.cdha.nshealth.ca/ 

http://www.ers.princeton.edu/index.html 

http://www.hc-sc.gc.ca/ 

http://www.bristol.ac.uk/education/ 

http://www.cbe.ab.ca/ 
http://www.cew wisc.edu/ 

http://www.epsb.ca/ 
http:/ / cea-ace.ca/home.cfm 

http://mathema tica-mpr.com/ 

http://www.csse.ca/About/About.shtml 

http://www.fraserinstitute.org/ 

http://www.health.gov.bc.ca/socsec/contacts.html 

http://www.tdsb.on.ca/ 

http://rainbowschools.ca/ 
http://www.health.gov.on.ca/ 

http://www.mern.ca/ 

http://ww wsdl.org/ 

http://www.tvdsb.ca/ 

http:/ / spsd.sk.ca/ 

http://www.rbe.sk.ca/ 

http://www. voicetheunion.org.uk/ 

http://www.health.alberta.ca/ 

http://www.peel.edu.on.ca/ 

http://www.vsb.bc.ca/ 
http://www.yrdsb.edu.on.ca/ 

http://www.serha.ca/serha/careers/index.php 

http://ies.ed.gov/ncee/wwc/ 

http://www.edu.pe.ca/finder/schoolwebpages.asp 

http://www.gov.ns.ca/health/ 
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Department of Education Science and Tech 
(Australia) 

Ministry of Education [MEXTj Japan 
Council of Ontario directors of Education 
Canadian centre for policy alternative 
American Federation of Teachers 
Phi Delta Kappan International 
Centre for Review and Dissemination 
OECD 

Australia Department of Education, 
Employment and Workplace Relations 
Human Resources and Social Development 
Canada 

Ministry of Education - Alberta 
Mid-continent Research for Education 
Learning 

Nova Scotia Teachers Union 

Canadian Association of Principals 

Newfoundland/Labrador - Education Ministry 

Learning First Alliance 

Ministry of Education - Saskatchewan 

Ministry of Education - Manitoba 

Ministry of Education Kingdom of Saudi 

Arabia 

Research in practice for Adults 
New Brunswick — Education Ministry 
Ministry of Education - British Columbia 
Nova Scotia Department of Education 
Chicago Public Schools 
San Francisco Unified School District 
Nunavut - Education Ministry 
Ministry of Education - Quebec 


http://www.dest.gov.au/default.htm 

http://www.mext.go.jp/ english/ 

http://www.ontariodirectors.ca/ 

http://www.policyalternatives.ca/ 

http://www.aft.org/ 
http://www.pdkintl.org/ 

http://www.york.ac.uk/inst/crd/about us.htm 

http://www.oecd.org/home/ 

http://www.deewr.gov.au/Pages/default.aspx 

http://www.hrsdc.gc.ca/eng/home.shtml 

http:/ / education.alberta.ca/ 

http://www.mcrel.org/ 

http://www.nstu.ca/app?service=page/Home 

http: / / www.cdnprincipals.org/index.html 

http://www.ed.gov.nl.ca/edu/ 

http://www.learningfirst.org/ 
http://www.education.gov.sk.ca/ 

http://www.edu.gov.mb.ca/ 

http://www.moe.gov.sa/openshare/englishcon/index.htm 

http://www.morgan-law.com/ripfa/ 

http://www.gnb.ca/OOOO/index-e.asp 

http://www.gov.bc.ca/bced/ 
http://www.ednet.ns.ca/index.shtml 

http://cps.edu/Pages/home.aspx 

http:/ / portal.sfusd.edu/template/default.cfm 

http://www.gov.nu.ca/ education/ 

http://www.meq.gouv.qc.ca/GR-PUB/m englis.htm 
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